Keystone corrections with quadrilateral objects

ABSTRACT

An example method includes receiving by a processor of an image capturing device, an image of a quadrilateral object captured by a camera of the image capturing device. A homography matrix is calculated by the processor based on the image of the quadrilateral object and a known dimension of the quadrilateral object. The processor performs a keystone correction based on the homography matrix to convert a subsequent image captured by the camera into a corrected image.

BACKGROUND

Cameras are used to capture images of various different scenes orobjects. Cameras capture images of scenes that are not perfectlyrectangular. Alternatively, cameras may be held or positioned such thatthey are not perfectly perpendicular to a horizontal centerline of ascene or object. As a result, the images may appear to be distorted whenthe images are captured and printed. For example, a square may appear asa trapezoid in a captured image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a side view of an example system forperforming keystone corrections with quadrilateral objects of thepresent disclosure;

FIG. 2 is a block diagram of an example image capturing device thatperforms keystone corrections with a quadrilateral object the presentdisclosure;

FIG. 3 is a diagram of example images used to calculate a homographymatrix;

FIG. 4 is a flow chart of an example method for performing a keystonecorrection with a quadrilateral object of the present disclosure; and

FIG. 5 is a block diagram of an example non-transitory computer readablemedium storing instructions executed by a processor of the presentdisclosure.

DETAILED DESCRIPTION

The present disclosure relates to performing keystone corrections ofimages captured by an image capturing device. Keystone correction may beused by image software to correct the distortions or skew of the images.For example, tall buildings may appear to be leaning to the right orleft when printed without keystone correction. The keystone correctionmay correct the image such that printed image appears to be a tallbuilding that is not leaning to either side.

To apply a correct amount of keystone correction, the camera may becalibrated. Some cameras may perform the calibration on each image.However, performing repeated calibrations can be a time consuming andprocessor intensive task. Other cameras may perform calibration on thefield of view rather than the image itself. Changing the field of viewmay be undesirable as the viewer may be distracted by the oddly shapedoutline of the field of view.

Examples described herein provide an approach to perform a calibrationto obtain a homography matrix. The homography matrix may then be appliedto perform a keystone correction on subsequently captured images. Thepresent disclosure performs the keystone correction on the image itselfand not the field of view. Thus, the images can be correctly displayedor printed on rectangular fields of view, or paper, without distortion.Moreover, the homography matrix may be used repeatedly as long as thecamera remains in the same position and viewing angle.

In an example, the calibration may be performed with any quadrilateralobject. For example, with the quadrilateral object may be a plain sheetof paper, or other relatively flat rectangular or square object, insteadof using a specialized mat.

FIG. 1 illustrates a block diagram of a system 100 for performingkeystone corrections with quadrilateral objects of the presentdisclosure. In one implementation, the system 100 may include an imagecapturing device 102, an endpoint device 104 and a display 106 that ison a surface 110 (e.g., a tabletop, desk, etc.). The image capturingdevice 102 may be a device that includes a camera or multiple types ofcameras to capture an image or different types of images with themultiple types of cameras. The image capturing device 102 may haveprocessing capabilities to perform keystone corrections on the image, asdescribed below. One camera may be a camera to capture still images orphotographs and the other types of cameras may be used to capturedimensional information (e.g., a three dimensional camera). The endpointdevice 104 may be a desktop computer, a laptop computer, a tabletcomputer, and the like. The display 106 may be a monitor, a television,and the like.

The image capturing device 102 may be in communication with the endpointdevice 104 to provide captured images to the endpoint device 104 fordisplay on the display 106, to be projected by an external imageprojector (not shown), to be printed on a printing apparatus (notshown), and the like. In one example, the image capturing device 102 maybe mounted onto the display 106 at a particular viewing angle 112 and aheight 114. However, due to the viewing angle 112 and the height 114 ofthe image capturing device 102, images that are captured by the imagecapturing device 102 may be distorted. For example, rectangular objectsmay appear as trapezoids in the captured image, or parallelograms,depending on how the rectangular object is orientated on the surface110.

In certain applications, it may be desirable to display a correctedimage. In other words, rectangular objects should appear as rectangularobjects and not trapezoids in a reproduced image of the image capturedby the image capturing device 102 and displayed in an unaltered field ofview.

In one example, a quadrilateral object 108 may be used to calibrate theimage capturing device 102. The calibration may generate a homographymatrix that can be used to perform keystone corrections on the objectswithin the image itself (as opposed to the viewing area to make theobject appear to be correctly oriented).

The quadrilateral object 108 may be any relatively flat object that liesas close to the same plane as the surface 110 as possible. For example,the quadrilateral object 108 may be fractions of millimeters thick. Thequadrilateral object 108 may be any rectangular or square object. In oneexample, the quadrilateral object 108 may be a sheet of paper (e.g., an8.5 inch×11 inch printing/copy paper).

It should be noted that the quadrilateral object 108 is not drawn toscale in FIG. 1. Although the quadrilateral object 108 is shown as beingrelatively thick and raised off of the surface 110 in FIG. 1 for ease ofillustration, the thickness of the quadrilateral object 108 may beindistinguishable from the surface 110. In other words, thequadrilateral object 108 and the surface 110 may appear to be on thesame horizontal plane.

FIG. 2 illustrates a block diagram of the image capturing device 102that performs the keystone corrections described herein. In one example,the image capturing device 102 may include a processor 202, a memory204, a red, green, blue (RGB) camera 208 and a three dimensional (3D)camera 210. The processor 202 may be in communication with the memory204, the RGB camera 208 and the 3D camera 210.

FIG. 3 illustrates example images used to generate a homography matrix206 stored in the memory 204 during the calibration process. The imagesillustrated in FIG. 3 may be referred to in conjunction with thedescription of the operation of the image capturing device 102.

In one example, the processor 202 may control the RGB camera 208 tocapture an image of the quadrilateral object 108. The quadrilateralobject 108 may be a rectangle (e.g., an 8.5 inch×11 inch sheet ofpaper). An example image 302 is illustrated in FIG. 3. The quadrilateralobject 108 may be placed on top of the surface 110 and within theviewing angle 112 of the image capturing device 102 such that the entirequadrilateral object 108 fits within a viewing area 300 of the image302. As noted above, in the present example, the quadrilateral object108 may be a rectangle but appear as a trapezoid in the image 302.

In one example, the processor 202 may control the 3D camera 210 tocapture an image of the quadrilateral object 108 to calculate thedimensions of the quadrilateral object 108. For example, the 3D camera210 may be a depth sensing camera or a stereoscopic camera that can useexisting software functions to measure the dimensions (e.g., a lengthand a width) of the quadrilateral object 108 based on the 3D image thatis captured.

In another example, a user may provide the dimensions to the imagecapturing device 102 via an interface of the endpoint device 104. Forexample, the user may enter the dimensions of a width of 8.5 inches anda length of 11 inches on the endpoint device 104. The endpoint device104 may then transmit the dimensions to the image capturing device 102.

The processor 202 may then generate a theoretical image 304 that wouldbe captured by a camera pointed perfectly perpendicular to a viewingplane of the quadrilateral object 108 using the known dimensions. Forexample, knowing that the dimensions of the quadrilateral object 108 are8.5 inches by 11 inches, the processor 202 may generate the theoreticalimage 304, which may be used as a second image to calculate an initialhomography matrix between the theoretical image 304 and the image 302.

The initial homography matrix may then be used to calculate an offset322 ₁, 322 ₂, 322 ₃, and 322 ₄ for each corner 320 ₁, 320 ₂, 320 ₃, and320 ₄, respectively, of the quadrilateral object 108. The offsets 322 ₁,322 ₂, 322 ₃, and 322 ₄ may provide values for how many pixels an objectmay be moved to reach an edge of the viewing area 300 based off an imagethat is corrected by the initial homography matrix. For example in thetheoretical image 304, the corner 320 ₁ may be set as coordinates (0,0). The offsets 322 ₁, 322 ₂, 322 ₃, and 322 ₄ for each corner 320 ₁,320 ₂, 320 ₃, and 320 ₄ may be calculated with reference to the corner320 ₁ at location (0, 0) and the values of the initial homographymatrix.

The offsets 322 ₁, 322 ₂, 322 ₃, and 322 ₄ may be applied to thequadrilateral object 108 in the theoretical image 304 to generate a fullframe image 306. For example, the quadrilateral object 108 may becorrected by the initial homography matrix and adjusted by the offsets322 ₁, 322 ₂, 322 ₃, and 322 ₄ to fill the entire viewing area 300. As aresult, the corrected image of the quadrilateral object 108 may appearto be rectangular rather than as a trapezoid in the image 302.

The processor 202 may then use the image 302 and the image 306 tocalculate the homography matrix 206 that is stored in the memory 204.The homography matrix 206 may then be applied to images subsequentlycaptured by the RGB camera 208. The homography matrix 206 may be used toperform keystone corrections on objects within the image within theviewing area 300, rather than correcting the viewing area 300 to makethe object appear to be corrected. Thus, the processor 202 may convert asubsequent image captured by the RGB camera 208 into a corrected imageby applying a keystone correction based on the homography matrix 206.

Furthermore, after the homography matrix 206 is calculated, thehomography matrix 206 may be repeatedly used. In other words, thecalibration process described may be performed once as long as the imagecapturing device 102 is not moved. In other words, as long as the imagecapturing device 102 maintains the same viewing angle 112 at the sameheight 114 for subsequently captured images, the same homography matrix206 may be used to perform keystone corrections on the subsequentlycaptured images.

In one example, the processor 202 may store the viewing angle 112 in thememory 204. Before an image is captured by the RGB camera 208, theprocessor 202 may compare a current viewing angle 112 to the storedviewing angle 112 to determine if the image capturing device 102 hasbeen moved. If the image capturing device 102 has been moved, theprocessor 202 may initiate the calibration process described above toupdate the values of the homography matrix 206 for the new viewing angle112. For example, an updated image of the quadrilateral object 108 maybe captured by the image capturing device 102 after being moved. Theupdated image may be used to calculate an updated homography matrix thatcan be used to perform keystone corrections.

FIG. 4 illustrates a flow diagram of an example method 400 forperforming keystone corrections with quadrilateral objects. In oneexample, the method 400 may be performed by the image capturing device102 or an apparatus 500 described below and illustrated in FIG. 5.

At block 402, the method 400 begins. At block 404, the method 400receives an image of a quadrilateral object captured by a camera of theimage capturing device. For example, an RGB camera of the imagecapturing device may capture an image of the quadrilateral object at aparticular viewing angle. The image of the quadrilateral object may bedistorted making the quadrilateral object appear as a trapezoid, orparallelogram, depending on how the quadrilateral object is orientedwithin the field of view of the image capturing device.

In one example, the quadrilateral object may be a relatively flatobject. For example, the quadrilateral object may be a sheet of paper,or other flat square or rectangular object.

At block 406, the method 400 calculates a homography matrix based on theimage of the quadrilateral object and a known dimension of thequadrilateral object. In one example, the dimensions (e.g., a length andwidth) of the quadrilateral image may be automatically calculated by animage captured by a 3D camera of the image capturing device. In anotherexample, the dimensions of the quadrilateral image may be provided by auser via a user interface of an endpoint device that is in communicationwith the image capturing device.

With the known dimensions, a theoretical image of the quadrilateralobject may be generated. For example, if the quadrilateral object was arectangular sheet of paper, the theoretical image would show arectangular sheet of paper having the known dimensions scaled relativeto the known dimensions. The theoretical image would appear as if acamera that is perpendicular to a horizontal plane of the rectangularsheet of paper captured the theoretical image. With the image capturedby the RGB camera in block 404 and the theoretical image, an initialhomography matrix may be calculated.

An offset for each corner of the quadrilateral object in the theoreticalimage may be calculated. For example, the top left corner of thequadrilateral object may be set with initial coordinates (0, 0). Theoffset may be calculated for each corner with reference to the initialcoordinates (0, 0) of the top left corner of the quadrilateral object.The offset may provide values with respect to how much each corner ofthe quadrilateral object that is corrected by the initial homographymatrix may be stretched to fill a field of view. The offsets may beapplied to the quadrilateral object to generate a full frame image ofthe quadrilateral object.

The full frame image of the quadrilateral object and the distorted imageof the quadrilateral object captured in block 404 may then be used togenerate a homography matrix that can be applied to subsequentlycaptured images. The homography matrix may be stored in memory and usedfor subsequent images captured by the RGB camera as long as the imagecapturing device is not moved.

At block 408, the method 400 performs a keystone correction based on thehomography matrix to convert a subsequent image captured by the camerainto a corrected image. For example, the image capturing device maycapture another picture with the RGB camera. The object within thesubsequent image may appear to be distorted. However, the homographymatrix that was previously calculated and stored in memory may beapplied to the subsequent image to perform a keystone correction andgenerate a corrected image.

In one example, the corrected image may have an unaltered field of viewand the object within the image may be corrected by the keystonecorrection. In other words, if the original field of view was arectangle and the object was a rectangle that appeared as a trapezoid inthe subsequent image, the corrected image may still have a field of viewthat is a rectangle and the object may appear as a rectangle instead ofa trapezoid. At block 410, the method 400 ends.

FIG. 5 illustrates an example of an apparatus 500. In one example, theapparatus 500 may be the image capturing device 102. In one example, theapparatus 500 may include a processor 502 and a non-transitory computerreadable storage medium 504. The non-transitory computer readablestorage medium 504 may include instructions 506, 508, 510 and 512 thatwhen executed by the processor 502, cause the processor 502 to performvarious functions.

In one example, the instructions 506 may include instructions to capturean image of a quadrilateral object in view of the image capturingdevice. The instructions 508 may include instructions to calculate ahomography matrix based on the image of the quadrilateral object and aknown dimension of the quadrilateral object. The instructions 510 mayinclude instructions to capture a subsequent image. The instructions 512may include instructions to perform a keystone correction on thesubsequent image based on the homography matrix to convert a subsequentimage captured by the image capturing device into a corrected image.

It will be appreciated that variants of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be combined intomany other different systems or applications. Various presentlyunforeseen or unanticipated alternatives, modifications, variations, orimprovements therein may be subsequently made by those skilled in theart which are also intended to be encompassed by the following claims.

The invention claimed is:
 1. A method, comprising: receiving, by aprocessor of an image capturing device, an image of a quadrilateralobject captured by a camera of the image capturing device; calculating,by the processor, a homography matrix based on the image of thequadrilateral object and a known dimension of the quadrilateral object,wherein the calculating the homography matrix comprises: calculating, bythe processor, an initial homography matrix based on the known dimensionof the quadrilateral object and the image; generating, by the processor,a rectangular image of the quadrilateral object using the initialhomography matrix; calculating, by the processor, a respective offset ofeach one of four corners of the rectangular image of the quadrilateralobject and the quadrilateral object in the image; and generating, by theprocessor, the homography matrix based on the respective offset of theeach one of the four corners; and performing, by the processor, akeystone correction based on the homography matrix to convert asubsequent image captured by the camera into a corrected image.
 2. Themethod of claim 1, wherein the known dimension comprises a length and awidth of the quadrilateral object.
 3. The method of claim 1, wherein theknown dimension is received via a user interface from a user.
 4. Themethod of claim 1, wherein the known dimension is detected based on athree dimensional camera of the image capturing device.
 5. The method ofclaim 1, wherein the receiving and the calculating are repeated when theimage capturing device is moved.
 6. A non-transitory computer readablestorage medium encoded with instructions executable by a processor ofimage capturing device, the non-transitory computer-readable storagemedium comprising: instructions to capture an image of a quadrilateralobject via the image capturing device; instructions to calculate ahomography matrix based on the image of the quadrilateral object and aknown dimension of the quadrilateral object; instructions to capture asubsequent image; and instructions to perform a keystone correction onthe subsequent image based on the homography matrix to convert asubsequent image captured by the image capturing device into a correctedimage, wherein the instructions to capture the image of thequadrilateral object and the instructions to calculate the homographymatrix are repeated when the image capturing device is moved.
 7. Thenon-transitory computer readable storage medium of claim 6, wherein theinstructions to calculate the homography matrix comprises: instructionsto calculate an initial homography matrix based on the known dimensionof the quadrilateral object and the image; instructions to generate arectangular image of the quadrilateral object using the initialhomography matrix; instructions to calculate a respective offset of eachone of four corners of the rectangular image of the quadrilateral objectand the quadrilateral object in the image; and instructions to generatethe homography matrix based on the respective offset of the each one ofthe four corners.
 8. The non-transitory computer readable storage mediumof claim 6, wherein the known dimension comprises a length and a widthof the quadrilateral object.
 9. The non-transitory computer readablestorage medium of claim 6, wherein the known dimension is received via auser interface from a user.
 10. The non-transitory computer readablestorage medium of claim 6, wherein the known dimension is detected basedon a three dimensional camera of the image capturing device.
 11. Anapparatus, comprising: a red, green, and blue (RGB) camera to capture animage of a quadrilateral object; a three dimensional (3D) camera todetect dimensions of the quadrilateral object; and a processor incommunication with the RGB camera and the 3D camera to: calculate ahomography matrix based on the image of the quadrilateral object and thedimensions of the quadrilateral object, perform a keystone correctionbased on the homography matrix to convert a subsequent image captured bythe RGB camera into a corrected image, detect movement of the apparatus,and calculate an updated homography matrix based on an updated image ofthe quadrilateral object and the dimensions of the quadrilateral object.12. The apparatus of claim 11, wherein to calculate the homographymatrix, the processor is to: calculate an initial homography matrixbased on the known dimension of the quadrilateral object and the image;generate a rectangular image of the quadrilateral object using theinitial homography matrix; calculate a respective offset of each one offour corners of the rectangular image of the quadrilateral object andthe quadrilateral object in the image; and generate the homographymatrix based on the respective offset of the each one of the fourcorners.